[1/3] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid]#35216
[1/3] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid]#35216damccorm merged 105 commits intoapache:masterfrom
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Thanks! It looks like there are a bunch of precommits failing with failures like: Would you mind taking a look? |
Co-authored-by: Danny McCormick <dannymccormick@google.com>
|
Assigning reviewers: R: @liferoad for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
waiting on author |
|
stop reviewer notifications |
|
Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment |
f29c481 to
dbfd5f5
Compare
60ef4a2 to
aa2231f
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #35216 +/- ##
============================================
+ Coverage 56.51% 56.52% +0.01%
Complexity 3319 3319
============================================
Files 1198 1199 +1
Lines 182870 183091 +221
Branches 3426 3426
============================================
+ Hits 103347 103496 +149
- Misses 76223 76295 +72
Partials 3300 3300
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Thanks @damccorm for taking another look at this PR. I've removed the explicit P.S. The failed |
|
Also, I plan to submit a follow-up PR perhaps after this one and that docs PR #35467? It will primarily address two pieces of feedback you provided:
|
damccorm
left a comment
There was a problem hiding this comment.
Thanks! This looks good to me
…ybrid] (apache#35216) * sdks/python: add pymilvus dependency * sdks/python: add `MilvusSearchEnrichmentHandler` * sdks/python: test `MilvusSearchEnrichmentHandler` * sdks/python: itest `MilvusSearchEnrichmentHandler` * examples: add `MilvusSearchEnrichmentHandler` * sdks/python: combine milvus search strategies in one Co-authored-by: Danny McCormick <dannymccormick@google.com> * sdks/python/container: update image requirements * sdks/python: add license for `milvus_search.py` * sdks/python: add docstrings for `milvus_search.py` * sdks/python: unit test milvus search handler * sdks/python: update docstrings for milvus_search * sdks/python: fix linting for `milvus_search.py` * sdks/python: add more unit tests for milvus search * sdks/python: combine test classes in one * sdks/python: add `setuptools` as dependency * sdks/python: update container image requirements * sdks/python: update definition for `ANNS` field * .github/workflows: upgrade pip & setuptools * sdks/python: fix linting issue for `milvus_search` * sdks/python: group I/O types together milvus_search * .github/workflows: upgrade pip & setuptools * .github: unify upgrading setuptools & pip * sdks/python: fix linting for `milvus_search.py` * sdks/python: update grpcio for py<=3.12 * sdks/python: update image requirements * sdks/python: add `milvus-lite` manual license * sdks/python: fix `milvus_search_it_test` failed cases * sdks/python: unify access to sparse/dense embeddings * sdks/python: invoke `unittest.main` on milvus search * sdks/pyhon: make `MilvusSearchDBContainerInfo` optional for linting * sdks/python+website: update docs * sdks/python: fix linting issues for `milvus_search` component * sdks/python: fix linting issues for milvus search component * website: add missing doc for milvus search * sdks/python: add itests for milvus search * sdks/python: complete itests for milvus search * sdks/python: fix linting * sdks/python: address (can be) flaky test cases * website: update relase version for `enrichment-milvus.md` * sdks/python: fix failed unit tests for milvus search * sdks/python: fix linting for milvus search itests * website: update docs html to ref milvus enrichment handler * sdks/python: avoid port collision for milvus container * sdks/python: remove free port allocation for milvus search * sdks/python: fix formatting issues for milvus search * sdks/python: fix linting for milvus_search_it_test * sdks/python: handle port collisions for milvus search itest * sdks/python: increase timeout for milvus container * sdks/python: experiment being explicit about the port solve the CI issue * sdks+.github: experiment running ml deps CI test onubuntu solve issue * .github/workflwos: revert python precommit ml changes * sdks/python: fix CI issues for itests * sdks/python: fix linting for milvus search itests * examples/notebook: update milvus enrichment transform * website: update milvus enrichment transform * CHANGES.md: add note for milvus enrichment handler * sdks/python: update itests for milvus search * sdks/python: fix linting issues * multi: update * multi: update * updatet * update * update * sdks/python: fix linting issues * sdks/python: see what CI workflows would fail * .github: run beam_PreCommit_Python_ML only on ubuntu-20.04 runner * .github: test workflow * .github: revert changes * .github: add milvus-integration-tests.yml * .github: update milvus it workflow * update * .github: update milvus-tests workflow * .github: try to use ubuntu version `ubuntu-20.04` * .github+sdks/python: update itests * .github: update gh runner for milvus itests * .github: update milvus itests workflow * .github+sdks/python: update itests * .github: remove `milvus-integration-tests.yml` for the PR review * sdks/python: skip itests properly if milvus db container failed to start * skds/python: restructure the code order in the example * sdks/python: reduce number of retries to avoid test timeout * sdks/python: set internal testcontainer env variable for max retries * sdks/python: update tc max retries * sdks/python: update * sdks/python: use dynamic milvus service and healthcheck ports * sdks/python: fix linting issues for milvus search itest * sdks/python: fixing linting issues for milvus search itests * .github+sdks/python: reconfigure dependencies * sdks/python: address Danny's feedback (2) * examples/notebooks: update `milvus_enrichment_transform` * website+examples: remove non-functional docs/examples * website: revert updated `enrichment.md` * sdks/python: remove duplicated `HybridSearchParameters` * sdks/python: fix linting for milvus search * sdks/python: remove examples from this PR * .github/workflows: remove unnecesssary changes * CHANGES.md: undo the feature template * sdks/python: remove `pip` and `setuptools` as explicit dependency --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
…ybrid] (apache#35216) * sdks/python: add pymilvus dependency * sdks/python: add `MilvusSearchEnrichmentHandler` * sdks/python: test `MilvusSearchEnrichmentHandler` * sdks/python: itest `MilvusSearchEnrichmentHandler` * examples: add `MilvusSearchEnrichmentHandler` * sdks/python: combine milvus search strategies in one Co-authored-by: Danny McCormick <dannymccormick@google.com> * sdks/python/container: update image requirements * sdks/python: add license for `milvus_search.py` * sdks/python: add docstrings for `milvus_search.py` * sdks/python: unit test milvus search handler * sdks/python: update docstrings for milvus_search * sdks/python: fix linting for `milvus_search.py` * sdks/python: add more unit tests for milvus search * sdks/python: combine test classes in one * sdks/python: add `setuptools` as dependency * sdks/python: update container image requirements * sdks/python: update definition for `ANNS` field * .github/workflows: upgrade pip & setuptools * sdks/python: fix linting issue for `milvus_search` * sdks/python: group I/O types together milvus_search * .github/workflows: upgrade pip & setuptools * .github: unify upgrading setuptools & pip * sdks/python: fix linting for `milvus_search.py` * sdks/python: update grpcio for py<=3.12 * sdks/python: update image requirements * sdks/python: add `milvus-lite` manual license * sdks/python: fix `milvus_search_it_test` failed cases * sdks/python: unify access to sparse/dense embeddings * sdks/python: invoke `unittest.main` on milvus search * sdks/pyhon: make `MilvusSearchDBContainerInfo` optional for linting * sdks/python+website: update docs * sdks/python: fix linting issues for `milvus_search` component * sdks/python: fix linting issues for milvus search component * website: add missing doc for milvus search * sdks/python: add itests for milvus search * sdks/python: complete itests for milvus search * sdks/python: fix linting * sdks/python: address (can be) flaky test cases * website: update relase version for `enrichment-milvus.md` * sdks/python: fix failed unit tests for milvus search * sdks/python: fix linting for milvus search itests * website: update docs html to ref milvus enrichment handler * sdks/python: avoid port collision for milvus container * sdks/python: remove free port allocation for milvus search * sdks/python: fix formatting issues for milvus search * sdks/python: fix linting for milvus_search_it_test * sdks/python: handle port collisions for milvus search itest * sdks/python: increase timeout for milvus container * sdks/python: experiment being explicit about the port solve the CI issue * sdks+.github: experiment running ml deps CI test onubuntu solve issue * .github/workflwos: revert python precommit ml changes * sdks/python: fix CI issues for itests * sdks/python: fix linting for milvus search itests * examples/notebook: update milvus enrichment transform * website: update milvus enrichment transform * CHANGES.md: add note for milvus enrichment handler * sdks/python: update itests for milvus search * sdks/python: fix linting issues * multi: update * multi: update * updatet * update * update * sdks/python: fix linting issues * sdks/python: see what CI workflows would fail * .github: run beam_PreCommit_Python_ML only on ubuntu-20.04 runner * .github: test workflow * .github: revert changes * .github: add milvus-integration-tests.yml * .github: update milvus it workflow * update * .github: update milvus-tests workflow * .github: try to use ubuntu version `ubuntu-20.04` * .github+sdks/python: update itests * .github: update gh runner for milvus itests * .github: update milvus itests workflow * .github+sdks/python: update itests * .github: remove `milvus-integration-tests.yml` for the PR review * sdks/python: skip itests properly if milvus db container failed to start * skds/python: restructure the code order in the example * sdks/python: reduce number of retries to avoid test timeout * sdks/python: set internal testcontainer env variable for max retries * sdks/python: update tc max retries * sdks/python: update * sdks/python: use dynamic milvus service and healthcheck ports * sdks/python: fix linting issues for milvus search itest * sdks/python: fixing linting issues for milvus search itests * .github+sdks/python: reconfigure dependencies * sdks/python: address Danny's feedback (2) * examples/notebooks: update `milvus_enrichment_transform` * website+examples: remove non-functional docs/examples * website: revert updated `enrichment.md` * sdks/python: remove duplicated `HybridSearchParameters` * sdks/python: fix linting for milvus search * sdks/python: remove examples from this PR * .github/workflows: remove unnecesssary changes * CHANGES.md: undo the feature template * sdks/python: remove `pip` and `setuptools` as explicit dependency --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
Description
Towards #35046.
Next #35577.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.